C++ template meta version of greatest common divisor and least common multiple.

I found these functions very useful often because it doesn't use any run-time computation cost. Calculations are done at compile time and intermediate values are not stored into the binary file.
It may increase compile time a lot if you try to get result with big numbers and only constant values can be used; no variables are allowed.

It is the functions:

template< int lhs, int rhs >
struct GreatestCommonDivisor
{
    enum { value = GreatestCommonDivisor< rhs, lhs % rhs >::value };
};

template< int lhs >
struct GreatestCommonDivisor< lhs, 0 >
{
    enum { value = lhs };
};

template< int lhs, int rhs >
struct LeastCommonMultiple
{
    enum { value = ( lhs * rhs ) / GreatestCommonDivisor< lhs, rhs >::value };
};

Here is how to use them:

int a = GreatestCommonDivisor< 24, 21 >::value;
int b = LeastCommonMultiple< 24, 21 >::value;

I hope this can help anybody who wants to improve the run-time performance.

Antlr grammar for C/C++ Preprocessor.

I made a Lexer/Parser/Application of C-preprocessor with Antlr and C#. I like to share my work with anybody who may have interest in doing it by himself or herself. *PS: Antlr is a parser generator.

Of course, there will be many or better ways to do the same thing than the way I did. Here is a summary of how I did.

1. Lexer A and Parser A take a C/C++ source text string and parse macro keywords such as #include, #define, #if, #ifdef, #ifndef and so on.
2. Application A stores #define keywords parsed from Parser A; there are two types of macro defines: one that takes arguments and another that doesn't take arguments.
3. Whenever #if or #elif are encountered, the macro keywords in the condition statement are resolved with the stored define macros. This process is done by Lexer A, Parser B; Lexer A is reused. This step is repeated until there is nothing to be replaced. Once all of macro keywords are resolved, arithmetic calculation is performed with Lexer B, Parser C and Application B.

Let me show you an example of #define and #if statements.

line1: #define VariableType 4
line2: #define FunctionType( a, b )     a - VariableType * b
line3:
line4: #if FunctionType( (2*3), 2 ) + 1 * 2
line5: #error "This should not be processed"
line6: #else
line7: #undef VariableType
line8: #endif

In the line1, a new macro keyword, "VariableType", will be stored by ApplicationA.

In the line2, a new macro keyword, "FunctionType", will be stored by input argument information.

In the line4, the macro, "FunctionType( (2*3), 2 )", is replaced by the definition and it will result in a new string, "(2*3) - VariableType * 2". The replacing step must be done after proper parser and that's why I needed a second Parser B for the macro resolving. For example, without Parser B, simple string replacement will cause errors in cases like "FunctionType( FunctionType( 1, 3 ), 3 )".

Since a macro definition can include another macro define keywords, one time macro resolving is not enough so the step had to be repeated until there is no macro keyword left.

Once all of macro keywords are resolved, the calculation must be performed based on operator precedence. That's where I needed 3rd Parser C and second Lexer B. For the example, the parser will parse a statement, "(2*3) - 4 * 2 + 1 * 2". This is a relatively simple calculator, which is Application B.

Based on the calculation result, one of statement blocks, which is line 5 or line 7, are ignored and another statement is processed. In the example, the calculation result is 0 and line7 will be processed.

I can share the Antlr grammars here but application source codes are too long to be posted; I wrote C# applications.

Parser A, which I call CPreprocessor, is here:

grammar CPreprocessor;
import CPreprocessorLexerRule;
// parser rules must start with lower case.

@parser::members
{
    protected const int EOF = Eof;
}

program        :    statement*    EOF    ;

statement    :    preprocess
            |    ( COMMENT_BLOCK | COMMENT_LINE )
            |    STRING                        // ignore
            |    ( ppChars | NEWLINE )        // ignore
            ;

// pp means preprocess
preprocess    :    ppInclude
            |    ppDefineFunc    // this must be before ppDefineVar: eg. #define AAA  ( a ) 
            |    ppDefineVar
            |    ppUndef
            |    ppPragma
            |    ppError
            |    ppIfStatement
            ;

ppChars        :    ( STRING | WS | PREPROCESS_BEGIN | DEFINED | ID | COMMA | PAREN_OPEN | PAREN_CLOSE | CHAR )+    ;

ppInclude    :    PREPROCESS_BEGIN    WS*    PREPROCESS_INCLUDE    WS*    ppChars    WS*    ( NEWLINE | EOF )    ;
ppPragma    :    PREPROCESS_BEGIN    WS*    PREPROCESS_PRAGMA    WS*    ppChars    WS*    ( NEWLINE | EOF )    ;
ppError        :    PREPROCESS_BEGIN    WS*    PREPROCESS_ERROR    WS*    ppChars    WS*    ( NEWLINE | EOF )    ;

ppUndef            :    PREPROCESS_BEGIN    WS*    PREPROCESS_UNDEF    WS+    ID                            WS*    ( NEWLINE | EOF )    ;
ppDefineVar        :    PREPROCESS_BEGIN    WS*    PREPROCESS_DEFINE    WS+    ID        ( WS+    ppChars )?    WS*    ( NEWLINE | EOF )    ;
ppDefineFunc    :    PREPROCESS_BEGIN    WS*    PREPROCESS_DEFINE    WS+    ppdfId    WS*    ppdfChars        WS*    ( NEWLINE | EOF )    ;

// ppdfi means preprocess define function ID
ppdfId            :    ID    PAREN_OPEN    WS*    ppdfiArguments    WS*    PAREN_CLOSE    ;    // WS between ID and '(' is not allowed.
ppdfiArguments    :    ppdfiArgument    ( WS*    COMMA    WS*    ppdfiArgument    )*    ;    // at least 1 argument is required.
ppdfiArgument    :    ID    ;

// ppdfc means preprocess define function characters
ppdfChars    :    ( ppdfcId | ppdfcNotId )+    ;
ppdfcId        :    ID    ;
ppdfcNotId    :    ( STRING | WS | PREPROCESS_BEGIN | DEFINED | COMMA | PAREN_OPEN | PAREN_CLOSE | CHAR )    ;


ppIfStatement    :    ( ppisIF | ppisIfDef | ppisIfNdef )    ;

ppisIF            :    PREPROCESS_BEGIN    WS*    PREPROCESS_IF        WS+    ppChars    WS*    NEWLINE            ppisStatement    ppisElifElseEndif    ;
ppisElif        :    PREPROCESS_BEGIN    WS*    PREPROCESS_ELIF        WS+    ppChars    WS*    NEWLINE            ppisStatement    ppisElifElseEndif    ;
ppisElse        :    PREPROCESS_BEGIN    WS*    PREPROCESS_ELSE                    WS*    NEWLINE            ppisStatement    ppisEndif    ;
ppisEndif        :    PREPROCESS_BEGIN    WS*    PREPROCESS_ENDIF                WS*    (NEWLINE|EOF)    ;
ppisIfDef        :    PREPROCESS_BEGIN    WS*    PREPROCESS_IFDEF    WS+    ID        WS*    NEWLINE            ppisStatement    ppisElseEndif    ;
ppisIfNdef        :    PREPROCESS_BEGIN    WS*    PREPROCESS_IFNDEF    WS+    ID        WS*    NEWLINE            ppisStatement    ppisElseEndif    ;

ppisStatement        :    statement*    ;
ppisElifElseEndif    :    ppisElif
                    |    ppisElseEndif
                    ;
ppisElseEndif        :    ppisElse
                    |    ppisEndif
                    ;

Lexer A, which I call "CPreprocessorLexerRule", is here:

lexer grammar CPreprocessorLexerRule;
// lexer rules must start with upper case.

@lexer::members {
    public const int CHANNEL_COMMENT_BLOCK = 1;
    public const int CHANNEL_COMMENT_LINE = 2;
    protected const int EOF = Eof;
    protected const int HIDDEN = Hidden;
}

//=======================================================================
COMMENT_BLOCK        :    '/*'    .*?        '*/'            -> channel(CHANNEL_COMMENT_BLOCK) ;
COMMENT_LINE        :    '//'    NOT_NL*                    -> channel(CHANNEL_COMMENT_LINE) ;
NL_ESC                :    '\\'    NL                        -> skip ;
STRING                :    '"'    (STRING_ESC|.)*?    '"'        ;
WS                    :    ' '    |    '\t'                    ;    // preprocessor cannot "skip" white space, because wb matters in several cases.
NEWLINE                :    NL                                ;    // preprocessor cannot "skip" new line, because wb matters in several cases.
PREPROCESS_BEGIN    :    '#'                                ;

//=======================================================================
// nmode doesn't seem to work for C#
// mode preprocessMode;
//=======================================================================
PREPROCESS_INCLUDE    :    'include'    ;
PREPROCESS_DEFINE    :    'define'    ;
PREPROCESS_UNDEF    :    'undef'        ;
PREPROCESS_IFDEF    :    'ifdef'        ;    // this must appear before 'if'
PREPROCESS_IFNDEF    :    'ifndef'    ;    // this must appear before 'if'
PREPROCESS_IF        :    'if'        ;
PREPROCESS_ELSE        :    'else'        ;
PREPROCESS_ELIF        :    'elif'        ;
PREPROCESS_ENDIF    :    'endif'        ;
PREPROCESS_PRAGMA    :    'pragma'    ;
PREPROCESS_ERROR    :    'error'        ;

//=======================================================================
// These must be below reserved keywords
//=======================================================================
DEFINED        :    'defined'    ;
ID            :    ID_LETTER    (ID_LETTER|DIGIT)*    ;
COMMA        :    ','                                ;
PAREN_OPEN    :    '('                                ;
PAREN_CLOSE    :    ')'                                ;
CHAR        :    '\''    .    '\''
            |    .                                // the rest of all
            ;

//=======================================================================
// fragment can be referenced only by lexer.
// fragment itself is not lexer rule so that it cannot be used by parser.
//=======================================================================
fragment ID_LETTER    :    [a-zA-Z_]            ;
fragment DIGIT        :    [0-9]                ;
fragment STRING_ESC    :    '\\'    [btnr0"\\]    ; // \b, \t, \n etc...
fragment NL            :    '\r'?    '\n'        ;
fragment NOT_NL        :    ~[\r\n]                ;

Parser B, which I call "CPMacroResolve", is here:

grammar CPMacroResolve;
import CPreprocessorLexerRule;

// Asttumptions for the input statement:
// 1. No block/line comment exists.
// 2. No '#' character exists.
// 3. No preprocessor keywords exists except "defined"

statement    :    sToken+    ;

sToken        :    stDefined
            |    stFunctionCall
            |    stVariable
            |    stOther
            ;

stDefined    :    DEFINED    WS+    stdId    ;
stdId        :    PAREN_OPEN    WS*    stdId    WS*    PAREN_CLOSE
            |    stdiId
            ;
stdiId        :    ID    ;

stVariable    :    ID    ;

stFunctionCall    :    ID    WS*    PAREN_OPEN    stfcArguments    PAREN_CLOSE    ;
stfcArguments    :    stfcaArgument    ( COMMA    stfcaArgument )*    ;
stfcaArgument    :    ( PAREN_OPEN    stfcaArgument    PAREN_CLOSE | STRING | WS | ID | CHAR )*    ;

stOther        :    ( STRING | WS | COMMA | PAREN_OPEN | PAREN_CLOSE | CHAR )    ;

Parser C, which I call "CPCondition", is here:

grammar CPCondition;
import CPConditionLexerRule;

// Assumptions:
// 1. Lexer will remove white spaces.

condition    : expression ;

// Indirect left-recursion for binary/ternary expressions is solved by "direct" left-recursion
expression    :    PAREN_OPEN    expression    PAREN_CLOSE                                                    # ceParen
            |    unaryExpression                                                                        # ceUnary
            |    expression    op=(AOP_MUL|AOP_DIV|AOP_MOD)        expression                            # ceMulDivMod
            |    expression    op=(AOP_ADD|AOP_SUB)                expression                            # ceAddSub
            |    expression    op=(BOP_SHL|BOP_SHR)                expression                            # ceShlShr
            |    expression    op=(CMP_LE|CMP_LT|CMP_GE|CMP_GT)    expression                            # ceCmpLeLtGeGt
            |    expression    op=(CMP_EQ|CMP_NE)                    expression                            # ceCmpEqNe
            |    expression    BOP_AND                                expression                            # ceBitAnd
            |    expression    BOP_XOR                                expression                            # ceBitXor
            |    expression    BOP_OR                                expression                            # ceBitOr
            |    expression    LOP_AND                                expression                            # ceLogicAnd
            |    expression    LOP_OR                                expression                            # ceLogicOr
            |    expression    TER_IF                                expression    TER_ELS    expression        # ceTerIf
            |    value                                                                                # ceValue
            ;

unaryExpression    :    op=(AOP_ADD|AOP_SUB|BOP_NOT|LOP_NOT)    expression    ;

value    :    val=FLOAT
        |    val=INT
        |    val=HEX
        |    val=OCT
        |    val=TRUE
        |    val=FALSE
        |    val=ID    // error
        ;

Lexer B, which I call "CPConditionLexerRule", is here:

lexer grammar CPConditionLexerRule;

// Logic operator
LOP_AND    :    '&&'    ;
LOP_OR    :    '||'    ;
LOP_NOT    :    '!'    ;

// Bit operator
BOP_NOT    :    '~'    ;
BOP_XOR    :    '^'    ;
BOP_AND    :    '&'    ;
BOP_OR    :    '|'    ;
BOP_SHL    :    '<<'    ;
BOP_SHR    :    '>>'    ;

// Arithmatic operator
AOP_ADD    :    '+'    ;
AOP_SUB    :    '-'    ;
AOP_MUL    :    '*'    ;
AOP_DIV    :    '/'    ;
AOP_MOD    :    '%'    ;

// Comparison operator
CMP_EQ    :    '=='    ;
CMP_NE    :    '!='    ;
CMP_LE    :    '<='    ;
CMP_LT    :    '<'    ;
CMP_GE    :    '>='    ;
CMP_GT    :    '>'    ;

// Ternary conditional
TER_IF    :    '?'    ;
TER_ELS    :    ':'    ;

// Parentheses
PAREN_OPEN    :    '('    ;
PAREN_CLOSE    :    ')'    ;

// Reserved keyword
DEFINED    :    'defined'    ;
TRUE    :    'true'    ;
FALSE    :    'false'    ;

// typed values
FLOAT    :    DIGIT+    '.'    DIGIT*    'f'?            // match 1. 39. 3.14159 etc...
        |            '.'    DIGIT+    'f'?            // match .1 .14159
        ;
INT        :    [1-9]    DIGIT*
        |    '0'
        ;
HEX        :    '0x'    DIGIT+    ;    // 0x000
OCT        :    '0'        DIGIT+    ;    // 0123
ID        :    ID_LETTER    (ID_LETTER|DIGIT)*    ;

// skip
NL_ESC    :    '\\'    '\r'?    '\n'    -> skip ;
WS        :    [ \t]+                    -> skip ;


// fragments
fragment ID_LETTER    :    [a-zA-Z_]    ;
fragment DIGIT        :    [0-9]    ;

I am hoping that this information can be helpful for anybody who is planning to make their own version of C preprocessor or any kind of preprocessors.

*PS: I read one IBM guy suggested that all of C preprocessing should be done by Lexer only. I don't see if it is a better way or not because as I described, C preprocessor also need parsing steps several times.

Compare modified time of two files on DOS batch file.

I made a little complex but useful batch function for comparing the modified timestamp of two files. It is DOS batch command. BTW there is no command that does this on batch command without help of any external utilities.

The way it works is get DIR result and parse date/time info. I convert year/month/day/hour/min into one big comparable integer value. Two timestamp values are subtracted and if the result is negative, it means the second is bigger value. For example:

SET /A C=%A% - %B%
IF "%C%" == "-" ECHO B is bigger

Here it goes:

:COMPARE_TIMESTAMP_GT
REM %~1 is first file name that needs to be compared to
REM %~2 is second file name that needs to be compared to
REM %~3 is return variable name; if "%~1 > %~2" is true in terms of timestamp, it returns 1; otherwise 0.
IF NOT EXIST %~1 GOTO :EOF
IF NOT EXIST %~2 GOTO :EOF
IF "%~3" == "" GOTO :EOF

FOR /F "TOKENS=1,2,3" %%A IN ('DIR /A-D %~1 ^| FIND "/"' ) DO SET DATE1=%%A %%B %%C
FOR /F "TOKENS=1,2,3" %%A IN ('DIR /A-D %~2 ^| FIND "/"' ) DO SET DATE2=%%A %%B %%C

SET /A TIMESTAMP1=%DATE1:~8,1%*60*24*31*12*10+%DATE1:~9,1%*60*24*31*12+%DATE1:~0,1%*60*24*31*10+%DATE1:~1,1%*60*24*31+%DATE1:~3,1%*60*24*10+%DATE1:~4,1%*60*24+%DATE1:~11,1%*60*10+%DATE1:~12,1%*60+%DATE1:~14,1%*10+%DATE1:~15,1%
SET /A TIMESTAMP2=%DATE2:~8,1%*60*24*31*12*10+%DATE2:~9,1%*60*24*31*12+%DATE2:~0,1%*60*24*31*10+%DATE2:~1,1%*60*24*31+%DATE2:~3,1%*60*24*10+%DATE2:~4,1%*60*24+%DATE2:~11,1%*60*10+%DATE2:~12,1%*60+%DATE2:~14,1%*10+%DATE2:~15,1%
IF "%DATE1:~17,1%" == "P" SET /A TIMESTAMP1=%TIMESTAMP1% +12*60
IF "%DATE2:~17,1%" == "P" SET /A TIMESTAMP2=%TIMESTAMP2% +12*60

SET /A TIMEDIFF=%TIMESTAMP1% - %TIMESTAMP2%

SET "%~3=1"
IF "%TIMEDIFF:~0,1%" == "-" SET "%~3=0"
IF "%TIMEDIFF%" == "0" SET "%~3=0"
GOTO :EOF

And here is how you use it:

CALL :COMPARE_TIMESTAMP_GT a.txt b.txt timediff
IF "%timediff%" ==  "1" ECHO a.txt is newer than b.txt
IF "%timediff%" ==  "0" ECHO b.txt is newer than a.txt or their timestamp is same.

I found this very useful so far. If anyone find bugs, you are welcome to suggest in comment section below.

Antlr for C# target generates XXXLexer.java file instead of XXXLexer.cs

I encountered an issue that Antrl4 generates java source code of Lexer instead of c-sharp source code. It turned out that it is a bug in the Antlr4 tool.

Instead of using "options{ language=CSharp_v4_5 }", the target language name should be passed into the tool as an input argument: http://stackoverflow.com/questions/17873164/antlr4-generating-lexer-in-java-instead-of-c-sharp.

java org.antlr.v4.Tool %* -Dlanguage=CSharp_v4_5

With the argument, it generated .cs version of Lexer.

Shell script: Converting DTS/AC3 audio to AAC

I have been using HandBrake to transcode movie files for reasons. One of the reasons was the audio codecs. iDevices and Raspberry Pi cannot perform hardware accelerated DTS/AC3 audio decoding due to license issues.

With HandBrake, I couldn't find a way to convert only audio but not video. What I wanted was not touching the video for quality reason and convert only the audio to AAC or MP3.

I found several people posted how to do that for MKV container format. From those scripts I made my own script that converts audio tracks. I don't know how to manipulate AVI or other container formats yet but most of cases MKV seems to be enough.

This script is faster than transcoding both video and audio because this converts only audio not video. And since it doesn't touch the video, the video quality wouldn't be affected as well.

I believe as long as you use Raspberry Pi, you wouldn't need to transcode video. But if you are using iDevices, you may need to transcode audio and video both.

You will need to install two packages.

sudo apt-get install mkvtoolnix ffmpeg

Now here is the script:

#!/bin/sh
mkv_input=$1
if [ ! -f "$mkv_input" ]; then
        echo "*** File not found: $mkv_input ***"
fi

DIRNAME=/usr/bin/dirname
BASENAME=/usr/bin/basename
WC=/usr/bin/wc
DATE=/bin/date
GREP=/bin/grep
HEAD=/usr/bin/head
TAIL=/usr/bin/tail
CUT=/usr/bin/cut
AVCONV=/usr/bin/avconv
MKVINFO=/usr/bin/mkvinfo
MKVEXTRACT=/usr/bin/mkvextract
MKVMERGE=/usr/bin/mkvmerge

filename_only=`$BASENAME "$mkv_input" .mkv`
dirname_only=`$DIRNAME "$mkv_input"`

trackCount=`$MKVINFO "$mkv_input" | $GREP track\ ID | $WC -l`
echo "[`$DATE`] $trackCount tracks are found from $mkv_input..."
echo "[`$DATE`] This script will generate temporary files on the way and you may want to remove them by yourself..."

trackId=0
while [ $trackId -lt $trackCount ]
do
        trackNum=`expr $trackId + 1`
        mkv_output=$dirname_only/$filename_only.$trackId.mkv
        audio_org=$dirname_only/_tmp_audio.$trackId.org
        audio_new=$dirname_only/_tmp_audio.$trackId.aac

        if [ -f "$mkv_output" ]; then
                echo "[`$DATE`] Already converted file found: $mkv_output"
                trackId=`expr $trackId + 1`
                continue;
        fi

        srcAudio=`$MKVINFO "$mkv_input" | $GREP Codec\ ID | $HEAD -$trackNum | $TAIL -1 | $GREP -e "\(DTS\|AC3\)" | $CUT -d ":" -f 2`
        if [ "$srcAudio" = "" ]; then
                trackId=`expr $trackId + 1`
                continue
        fi
        echo "[`$DATE`] Audio track found at $trackId: $srcAudio"

        echo "[`$DATE`] Extracting audio to a file: $audio_org ..."
        $MKVEXTRACT tracks "$mkv_input" $trackId:"$audio_org"
        if [ ! -f "$audio_org" ]; then
                echo "[`$DATE`] *** Audio track extraction failed ***"
                trackId=`expr $trackId + 1`
                continue
        fi

        echo "[`$DATE`] Converting extracted audio to AAC: $audio_new ..."
        $AVCONV -i "$audio_org" -strict experimental -acodec 'aac' -ac '2' -pass '1' -y "$audio_new"
        if [ ! -f "$audio_new" ]; then
                echo "[`$DATE`] *** Audio converting failed ***"
                trackId=`expr $trackId + 1`
                continue
        fi

        echo "[`$DATE`] Merging new audio into MKV: $mkv_output ..."
        $MKVMERGE -o "$mkv_output" -A "$mkv_input" "$audio_new"
        if [ ! -f "$mkv_output" ]; then
                echo "[`$DATE`] *** Merging failed ***"
        fi

        trackId=`expr $trackId + 1`
done

echo "[`$DATE`] Done"

As you can see it generates temporary files on the way. You will need to manually remove them or you will need to modify the script by yourself. I just don't like any scripts to remove anything for safety reasons.

How to setup Raspberry Pi for Foscam IP Camera

I explained how to store video data from Foscam IP Camera on Raspberry Pi. Now I am going to show you how to wire them all together.

As I explained, Foscam IP Camera, FI8910W, has motion detection feature in it. When any motion is detected, it will either send an email or upload a few still images to FTP server. A problem is that it doesn't store video data. That's what I am trying to do in this article.

The trick is that on the camera side it will do the motion detection and uploading files to FTP. When FTP connection is requested, it actually interact with a fake ftp server and the fake ftp server will trigger the video recording with a script that I explained.

Here is a big picture of how whole thing works.

1. It opens a fake ftp port and wait for FTP request.
2. Camera request FTP login.
3. Start recording video stream
4. Wait for another FTP request to identify whether or not there is motion going on still.
5. If FTP access is request within a given time, it keeps recording the video stream.
6. If FTP access is not requested within a given time, it stops recording.

The setting on the camera side is already explained here.

A fake ftp server for shell script version is here:

#!/bin/sh
CUT=/usr/bin/cut
TR=/usr/bin/tr

if [ ! -x $CUT -o ! -x $TR ]; then
        echo "*** some of utilities are not found or not executable ***"
        exit 1
fi

echo "220 Welcome message."
read read_id
if [ ! "`echo $read_id | $CUT -d ' ' -f 1`" = "USER" ]; then
        echo "530 Invalid login information..."
        exit 2
fi
ID=`echo $read_id | $CUT -d ' ' -f 2 | $TR -d ^M`

echo "331 Please specify the password."
read read_pw
if [ ! "`echo $read_pw | $CUT -d ' ' -f 1`" = "PASS" ]; then
        echo "530 Invalid login information..."
        exit 3
fi
PW=`echo $read_pw | $CUT -d ' ' -f 2 | $TR -d ^M`

echo "230 Guest login ok, access restrictions apply."

read read_typei
if [ ! "`echo $read_typei | $TR -d ^M`" = "TYPE I" ]; then
        exit 4
fi
echo "200 Type Set to I"

echo ID=$ID >&2
echo PW=$PW >&2

I placed it here: /home/pi/script/foscam_fake_ftp.sh
But you can place it wherever you want.

In the script, a special character, "^M", is not two characters but one character. In VI, you can type in the character by pressing "Ctrl+V Ctrl+M". The special character was entailed when the camera send text message so I had to remove them in the script.

A difference compare to the previous Windows batch file version is that the ID and Password information given to the fake ftp server will be used as a login information to access the video data. This time, we don't care whether it is Passive or not.

Here is a configuration file that will be placed at /etc/foscam.conf

#!/bin/sh
fake_ftp_if=wlan0
fake_ftp_port=2122
foscam_fake_ftp=/home/pi/script/foscam/foscam_fake_ftp.sh
foscam_record=/home/pi/script/foscam/foscam_record.sh
foscam_listener=/home/pi/script/foscam/foscam_listener.sh
foscam_cron=/home/pi/script/foscam/foscam_cron.sh

record_basedir=/home/pi/camera
recording_ip_dir=/var/run/foscam.d
#recording_ip_dir=/home/pi/script/foscam/var

alarm_wait_interval=60
cron_sleep_interval=2
file_ext=.mkv

LISTENER_PIDFILE=/var/run/fld.pid
CRON_PIDFILE=/var/run/fcd.pid

WGET=/usr/bin/wget
GREP=/bin/grep
CUT=/usr/bin/cut
MKDIR=/bin/mkdir
DATE=/bin/date
GSTLAUNCH=/usr/bin/gst-launch-1.0

NCAT=/usr/bin/ncat
GREP=/bin/grep
CUT=/usr/bin/cut
SLEEP=/bin/sleep
PS=/bin/ps
KILL=/bin/kill
DATE=/bin/date
IFCONFIG=/sbin/ifconfig
TAIL=/usr/bin/tail
RM=/bin/rm

LS=/bin/ls

Now we need a listener script here:

#!/bin/sh
. /etc/foscam.conf

if [ "$foscam_fake_ftp" = "" -o ! -x $foscam_fake_ftp ]; then
        echo "*** fake ftp server is not found or executable: $foscam_fake_ftp ***"
        exit 0
fi
if [ "$foscam_record" = "" -o ! -x $foscam_record ]; then
        echo "*** fake ftp server is not found or executable: $foscam_record ***"
        exit 1
fi
if [ ! -x $NCAT -o ! -x $GREP -o ! -x $CUT -o ! -x $SLEEP -o ! -x $PS -o ! -x $KILL -o ! -x $DATE -o ! -x $IFCONFIG -o ! -x $TAIL -o ! -x $RM ]; then
        echo "*** some of utilities not found ***"
        exit 3
fi

fake_ftp_ip=`$IFCONFIG $fake_ftp_if | $GREP inet\ addr | $CUT -d ":" -f 2 | $CUT -d " " -f 1`
#echo fake_ftp_ip=$fake_ftp_ip

while [ true ]; do
        echo [`$DATE`] Waiting for a new request...
        ncat_result=`$NCAT -v -l -c $foscam_fake_ftp $fake_ftp_ip $fake_ftp_port 2>&1 3> /dev/null`
        if [ ! $? = 0 ]; then
                echo "*** Fake ftp cannot start: $fake_ftp_ip:$fake_ftp_port ***"
                exit 5
        fi

        #echo "$ncat_result"
        IP=`echo "$ncat_result" | $GREP "Ncat: Connection from " | $GREP ":[0-9]" | $CUT -d " " -f 4 | $CUT -d ":" -f 1`
        ID=`echo "$ncat_result" | $GREP ID= | $CUT -d "=" -f 2`
        PW=`echo "$ncat_result" | $GREP PW= | $CUT -d "=" -f 2`
        echo [`$DATE`] Streaming requested from $ID\@$IP...
        #echo PW=$PW

        info_file=$recording_ip_dir/$IP
        if [ ! -f $info_file ]; then
                record_result=`$foscam_record $IP $ID $PW`
                #echo "$record_result"

                alarm_upload_interval=`echo "$record_result" | $GREP alarm_upload_interval= | $CUT -d "=" -f 2`
                record_interval=`expr $alarm_upload_interval + $alarm_wait_interval`
                gstreamer_pid=`echo "$record_result" | $GREP pid= | $CUT -d "=" -f 2`

                echo "gstreamer_pid=$gstreamer_pid" > $info_file
                if [ ! -f $info_file ]; then
                        echo [`$DATE`] info file coundn\'t be created: $info_file
                        echo [`$DATE`] Recording streamming video from $ID\@$IP for $record_interval seconds...
                        $SLEEP $record_interval
                        $KILL $gstreamer_pid
                else
                        record_started=`$DATE +%s`
                        finish_recording_time=`expr $record_started + $record_interval`

                        echo "record_started=$record_started" >> $info_file
                        echo "alarm_upload_interval=$alarm_upload_interval" >> $info_file
                        echo "record_interval=$record_interval" >> $info_file
                        echo "finish_recording_time=$finish_recording_time" >> $info_file
                fi
        else
                more_recording_requested=`$DATE +%s`
                record_interval=`$GREP record_interval= $info_file | $CUT -d "=" -f 2`
                finish_recording_time=`expr $more_recording_requested + $record_interval`

                echo "more_recording_requested=$more_recording_requested" >> $info_file
                echo "finish_recording_time=$finish_recording_time" >> $info_file
        fi
done

I placed it at  /home/pi/script/foscam/foscam_listener.sh

In order to get this script working, you will need three other scripts: foscam_record.sh, foscam_cron.sh and foscam_fake_ftp.sh, which is shown above.

You can get foscam_record.sh from this previous article of mine. But you must get "gst-launch-1.0" running as a background process. It can be easily done by adding "&" at the end of the line.

Another script, foscam_cron.sh, will figure out when to stop recording. This script should be running all the time as a daemon or cron-job.

#!/bin/sh
. /etc/foscam.conf

if [ ! -d $recording_ip_dir ]; then
        echo "*** recording_ip_dir does not exist: $recording_ip_dir ***"
        exit 1
fi

if [ ! -x $GREP -o ! -x $LS -o ! -x $CUT -o ! -x $PS -o ! -x $RM -o ! -x $SLEEP -o ! -x $DATE -o ! -x $KILL ]; then
        echo "*** some of utilities not found ***"
        exit 3
fi

while [ true ]
do
        for info_filename in `$LS $recording_ip_dir`
        do
                info_file=$recording_ip_dir/$info_filename
                #echo info_file=$info_file

                gstreamer_pid=`$GREP gstreamer_pid= $info_file | $CUT -d "=" -f 2`
                if ! $PS $gstreamer_pid 2> /dev/null 1> /dev/null
                then
                        echo "*** gstreamer not found: $gstreamer_pid ***"
                        $RM $info_file
                        continue
                fi

                cur_time=`$DATE +%s`
                finish_recording_time=`$GREP finish_recording_time= $info_file | $TAIL -1 | $CUT -d "=" -f 2`
                if [ $finish_recording_time -lt $cur_time ]; then
                        echo Time to stop: $info_file ...
                        $KILL $gstreamer_pid
                        $RM $info_file
                fi
        done

        $SLEEP $cron_sleep_interval
done

Now I want it to start at boot time automatically.
You will have to make a new file at /etc/init.d/foscamd

#!/bin/sh

### BEGIN INIT INFO
# Provides:          wrice.blogspot.com
# Required-Start:    $network $local_fs $remote_fs
# Required-Stop:     $network $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: start Foscam daemons (fcd and fld)
### END INIT INFO

. /etc/foscam.conf

# See if the daemon is there
test -x $foscam_listener || exit 0

. /lib/lsb/init-functions

fake_ftp_ip=`$IFCONFIG $fake_ftp_if | $GREP inet\ addr | $CUT -d ":" -f 2 | $CUT -d " " -f 1`

remove_stale_pid_file () {
        pid_file=$1
        # Wait a little and remove stale PID file
        sleep 1
        if [ -f $pid_file ] && ! ps h `cat $pid_file` > /dev/null
        then
                rm -f $pid_file
        fi
}

case "$1" in
        start)
                log_daemon_msg "Starting Foscam daemons"

                log_progress_msg "fcd"
                if ! start-stop-daemon --start --quiet --background --make-pidfile --pidfile $CRON_PIDFILE --exec $foscam_cron
                then
                        log_end_msg 1
                        exit 1
                fi

                log_progress_msg "fld"
                if ! start-stop-daemon --start --quiet --background --make-pidfile --pidfile $LISTENER_PIDFILE --exec $foscam_listener
                then
                        log_end_msg 1
                        exit 1
                fi
                sleep 1
                if ! ps h `cat $LISTENER_PIDFILE` > /dev/null
                then
                        log_end_msg 1
                        exit 1
                fi
                log_end_msg 0
                ;;
        stop)
                end_msg=0
                log_daemon_msg "Stopping Foscam daemons"
                log_progress_msg "fld"
                if ! start-stop-daemon --stop --quiet --pidfile $LISTENER_PIDFILE
                then
                        end_msg=1
                fi
                echo | /usr/bin/ncat --send-only $fake_ftp_ip $fake_ftp_port
                remove_stale_pid_file $LISTENER_PIDFILE

                log_progress_msg "fcd"
                if ! start-stop-daemon --stop --quiet --pidfile $CRON_PIDFILE
                then
                        end_msg=1
                fi
                remove_stale_pid_file $CRON_PIDFILE
                log_end_msg $end_msg
                ;;
        reload|restart|force-reload)
                $0 stop
                sleep 1
                $0 start
                ;;
        status)
                status="0"
                status_of_proc -p $LISTENER_PIDFILE $foscam_listener foscamd || status=$?
                exit $status
                ;;
        *)
                echo "Usage: /etc/init.d/foscamd {start|stop|reload|restart|force-reload|status}"
                exit 1
                ;;
esac

exit 0

Once foscamd is created, you will need to run this command to generate symbolic links for each boot sequence:

sudo update-rc.d foscamd defaults

Now when you restart your computer the listener will start automatically.

Record Foscam IP Camera streaming video on Raspberry Pi

I spent whole day to implement this code. I think there will be many people who would like to use Raspberry Pi as a IP Camera recorder so I am sharing it here.

This is a shell script that uses GStream 1.0 library, which is not an official Wheezy dist yet. The reason why I am using it is that it was only one way to use hardware accelerated H.264 encoder. Currently only gstream-0.10 is officially available not 1.0 yet.

If you are not comfortable using unverified package, you can also use VLC without hardware acceleration. I haven't tested VLC way thoroughly but it shouldn't be too hard for anybody to use it.

Also my script uses "wget" to retrieve some information from Foscam IP Camera such as "alias name" and "alarm_upload_interval". If you don't want to install wget and you know those values for sure, you can simply modify the script and hard-code it.

#!/bin/sh
if [ ! $# = 3 ]; then
        echo "*** input argument error: IP ID PW ***"
        exit 1
fi
IP=$1
ID=$2
PW=$3

record_basedir=/home/pi/camera
file_ext=.mkv

WGET=/usr/bin/wget
GREP=/bin/grep
CUT=/usr/bin/cut
MKDIR=/bin/mkdir
DATE=/bin/date
GSTLAUNCH=/usr/bin/gst-launch-1.0

if [ ! -x $WGET -o ! -x $GREP -o ! -x $CUT -o ! -x $MKDIR -o ! -x $DATE -o ! -x $GSTLAUNCH ]; then
        echo "*** some of utilities not found ***"
        exit 2
fi

alias_name=`$WGET -q -S -O - http://$ID\@$IP/get_params.cgi\?user=$ID\&pwd=$PW 2> /dev/null | $GREP var\ alias= | $CUT -d"'" -f 2`
if [ "$alias_name" = "" ]; then
        echo "*** alias_name not found ***"
        exit 3
fi

record_dir=$record_basedir/$alias_name
if [ ! -d $record_dir ]; then
        $MKDIR -p $record_dir
fi
if [ ! -d $record_dir ]; then
        echo "*** Cannot make a folder: $record_dir ***"
        exit 4
fi

alarm_upload_interval=`$WGET -q -S -O - http://$ID\@$IP/get_params.cgi\?user=$ID\&pwd=$PW 2> /dev/null | $GREP var\ alarm_upload_interval= | $CUT -d"=" -f 2 | $CUT -d";" -f 1`
echo alarm_upload_interval=$alarm_upload_interval

timestamp=`$DATE +%Y_%m_%d-%H_%M_%S`
file_name=$timestamp$file_ext
fullpath=$record_dir/$file_name
echo file_name=$file_name
echo fullpath=$fullpath

$GSTLAUNCH souphttpsrc location="http://$ID\@$IP/videostream.asf\?user=$ID\&pwd=$PW" ! decodebin ! videoconvert ! omxh264enc ! "video/x-h264,profile=high" ! h264parse ! matroskamux ! filesink location=$fullpath 2> /dev/null 1> /dev/null

echo pid=$!

The last line is the core of the script. The logic is that it retrieve ASF video streaming data from Foscam IP Camera with "souphttpsrc". We need to decode the video with "decodebin". Then it becomes "raw video". The raw video is piped into "videoconvert". Now it is passed to the hardware accelerated H.264 encoder, "omxh264enc". I don't know about "h264parse" but without it, it didn't go through so you need the step as well. Then the encoded H.264 video is stored as MKV with "matroskamux" plug-in. Finally the file name for the output data is specified with "filesink" plug-in.

I found that two steps, souphttpsrc and decodebin, can be merged with "uridecodebin". It seems that uridecodebin can handle "buffering" feature and probably it would work better in different cases.

Another thing I want to mention is that stderr of gst-launch-1.0 should be redirected to /dev/null. Otherwise, it will make the script process "defunct". I think a child process of gst-launch-1.0 is holding the stderr of the parents' and it causes defunct processors when those parents are dead.


In order to install gstream1.0, you will need to follow these steps:

$ echo "deb http://vontaene.de/raspbian-updates/ . main" >> /etc/apt/sources.list
$ apt-get update
$ apt-get install libgstreamer1.0-0 libgstreamer1.0-0-dbg libgstreamer1.0-dev liborc-0.4-0 liborc-0.4-0-dbg liborc-0.4-dev liborc-0.4-doc gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer1.0-alsa gstreamer1.0-doc gstreamer1.0-omx gstreamer1.0-plugins-bad gstreamer1.0-plugins-bad-dbg gstreamer1.0-plugins-bad-doc gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-base-dbg gstreamer1.0-plugins-base-doc gstreamer1.0-plugins-good gstreamer1.0-plugins-good-dbg gstreamer1.0-plugins-good-doc gstreamer1.0-plugins-ugly gstreamer1.0-plugins-ugly-dbg gstreamer1.0-plugins-ugly-doc gstreamer1.0-pulseaudio gstreamer1.0-tools gstreamer1.0-x libgstreamer-plugins-bad1.0-0 libgstreamer-plugins-bad1.0-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev

Then you can verify that you have omxh264enc, which is the most important one, by this command:

$ gst-inspect-1.0 | grep omxh264enc
omx:  omxh264enc: OpenMAX H.264 Video Encoder

Now in case you want to use VLC, you can use this command:

vlc http://$ID\@$IP/videostream.asf\?user=$ID\&pwd=$PW\&res=8\&rate=6 --run-time=10 -Idummy --sout=#transcode{vcodec=h264}:standard{dst="a.mp4"} vlc://quit

I think it will be better to do -Irc without "--run-time=10 -Idummy" and control the time with remote control process. But I didn't go deep into the step.

As a reference of GStream, I found this page the most useful.

With more tweaks, I will be able to append time stamp on the corner of the video and I will also be able to include audio. But I haven't been gone that far yet.

Google AdSense rejected me for "copyright materials"

Today I remembered something that I was wanted to do before I come to America, which is 6 years ago. Web banner.

It was super easy to apply for Google AdSense; it took only 5 minutes to apply. And a few hours later, I got an email saying they rejected my application due to copyright material on my site.

First of all I was disappointed. I thought I can motivate my self for posting more quality articles on my blog by making some pennies. My excitement was changed to a big disappointment.

Second of all, I felt offended. I checked all the articles on my blog three times. Most of them are proudly my own thoughts or ideas. What they mean by "copyright materials" seems like some of illegal torrent link or video/audio files. I do have some video files uploaded but they are all my own videos.

It seems like it is not only me who had been rejected for the same reason. So I am feeling better after finding that there are more people who are like me.

I don't think there is any way to "prove" who owns a video file. I think they are trying to be risk-averse for unknown zones. None technical people may think my script files or game related posting some kind of "hacking" without knowing what those articles actually mean.


Actually, I think I am still feeling offended. But I guess that's what it is... No pennies for me from Google.

I was having a trouble connecting Wifi of my IP camera, FI8910W, to my new router, SBG6580.

I got the new router from TimeWarnerCable after I upgraded my Internet speed. The router was actually a good one; I like it a lot. It supports Dual band 5GHz 802.11n and it is a Modem and Wifi router combined device.

However, what I found is that my new router doesn't support old Wifi protocols, 802.11b/g. It only supports 802.11n. Also my IP Camera is not working well as 802.11n, although the tech spec stated "802.11b/g/n".

After wasting a few hours, I ended up using both Wifi routers together: one that covers 5GHz 802.11n and another that covers 802.11b/g/n. Since one of them is using a different frequency, 5GHz, I think they should work without much signal interfere.

Batch file that converts AC3/DTS/EAC3 movie files to MP3

I watch movies through iPad or iPhone. I connect them to TV so that I can enjoy the big screen on my sofa. The WiFi speed is fast enough to carry high definition movies but I found one small problem. Apple is banning AC3/EAC3/DTS movie players. Currently there is no movie play app on iOS that is able to play AC3/EAC3/DTS and access network shared video files.

I think it is some kind of license of patent issues that Apple is complying. The best way for me to work around is to convert those movie files to normal MP3 audio format. So I made a batch file that detect audio format and convert them if necessary.

Audio format can be detected by two softwares: ffmpeg and mplayer. I am using ffmpeg in this article.
There are several transcoding software but I decided to use HandBrake.

Here is my batch file. Save it as c:\batch\convert_audio.bat.

@ECHO OFF
SET ConvertedFolderName=Converted
SET ConvertedExt=mp4
SET VideoSpeed=medium

SET ffmpeg=C:\Utils\ffmpeg\ffmpeg-20130603-git-2976e2a-win32-static\bin\ffmpeg.exe
SET HandBrakeCLI=C:\Program Files\Handbrake\HandBrakeCLI.exe

REM Removing quotation marks from input file path...
SET InputFile=%*
SET InputFile=%InputFile:"=%
SET InputFile=%InputFile:"=%
IF "x%InputFile%" == "x=" SET InputFile=

IF NOT EXIST "%ffmpeg%" GOTO :ERROR_ARG
IF NOT EXIST "%HandBrakeCLI%" GOTO :ERROR_ARG
IF NOT EXIST "%InputFile%" GOTO :ERROR_ARG
REM SET DebugCmd=ECHO

ECHO === Checking audio type: %InputFile% ===
SET AudioProblemDetected=
FOR /F "usebackq tokens=*" %%A IN (`CALL "%ffmpeg%" -i "%InputFile%" 2^>^&1 ^| find "Audio: ac3" `) DO SET AudioProblemDetected=True
FOR /F "usebackq tokens=*" %%A IN (`CALL "%ffmpeg%" -i "%InputFile%" 2^>^&1 ^| find "Audio: dts" `) DO SET AudioProblemDetected=True
FOR /F "usebackq tokens=*" %%A IN (`CALL "%ffmpeg%" -i "%InputFile%" 2^>^&1 ^| find "Audio: eac3" `) DO SET AudioProblemDetected=True
IF "x%AudioProblemDetected%" == "x" GOTO :END
ECHO === Audio needs to be converted ===

SET DriveLetter=
SET BaseDir=
SET FileNameOnly=
FOR %%A IN ("%InputFile%") DO (
    SET DriveLetter=%%~dA
    SET BaseDir=%%~pA
    SET FileNameOnly=%%~nA
)

SET InputDir=%DriveLetter%%BaseDir%
IF NOT EXIST "%InputDir%" GOTO :ERROR_PARSE

SET OutputDir=%InputDir%%ConvertedFolderName%
IF NOT EXIST "%OutputDir%" MKDIR "%OutputDir%"
IF NOT EXIST "%OutputDir%" GOTO :ERROR_OUTPUT_DIR

SET OutputFile=%OutputDir%\%FileNameOnly%.%ConvertedExt%
IF EXIST "%OutputFile%" GOTO :ERROR_OUTPUT_FILE_EXIST

ECHO === [%DATE% %TIME%] Converting file to %OutputFile% ... ===
%DebugCmd% "%HandBrakeCLI%" -i "%InputFile%" -o "%OutputFile%" --encoder x264 --x264-preset %VideoSpeed% --aencoder faac --rate 29.97 --pfr
IF NOT EXIST "%OutputFile%" GOTO :ERROR_FAILED_HANDBRAKE

ECHO === [%Date% %TIME%] Convertion finished ===
GOTO :END

:ERROR_FAILED_HANDBRAKE
ECHO *** It seems like HandBrake didn't generate the file ***
GOTO :ERROR_OUTPUT_FILE_EXIST_DUMP

:ERROR_OUTPUT_FILE_EXIST
ECHO *** Output file already exists ***
:ERROR_OUTPUT_FILE_EXIST_DUMP
ECHO OutputFile = %OutputFile%
GOTO :ERROR_OUTPUT_DIR_DUMP

:ERROR_OUTPUT_DIR
ECHO *** Output directory cannot be created ***
:ERROR_OUTPUT_DIR_DUMP
ECHO OutputDir = %OutputDir%
GOTO :ERROR_PARSE_DUMP

:ERROR_PARSE
ECHO *** Parsing error ***
:ERROR_PARSE_DUMP
ECHO Drive letter = %DriveLetter%
ECHO Folder name = %BaseDir%
ECHO File name = %FileNameOnly%
ECHO InputDir = %InputDir%
GOTO :ERROR_ARG_DUMP

:ERROR_ARG
ECHO *** Incorrect arguments ***
:ERROR_ARG_DUMP
ECHO ffmpeg = %ffmpeg%
ECHO HandBrakeCLI = %HandBrakeCLI%
ECHO InputFile = %InputFile%
GOTO :END

:END

As an optional step, I also made another batch file that takes a folder name and convert every files under the subfolders. Save it as c:\batch\convert_audio_folder.bat.

@ECHO OFF
SET CONVERT_AUDIO_FILE=C:\batch\convert_audio.bat

SET BaseDir=%*
SET BaseDir=%BaseDir:"=%
SET BaseDir=%BaseDir:"=%
REM SET DebugCmd=ECHO

IF NOT EXIST "%CONVERT_AUDIO_FILE%" GOTO :ERROR_ARG
IF "x%BaseDir%" == "x=" SET BaseDir=
IF NOT EXIST "%BaseDir%" GOTO :ERROR_ARG

FOR /F "tokens=*" %%A IN ('dir /b /s /a-d "%BaseDir%"') DO @%DebugCmd% %CONVERT_AUDIO_FILE% %%A

GOTO :END

:ERROR_ARG
ECHO *** Incorrect arguments ***
:ERROR_ARG_DUMP
ECHO CONVERT_AUDIO_FILE = %CONVERT_AUDIO_FILE%
ECHO BaseDir = %BaseDir%
GOTO :END

:END

This converting seems to take forever and I will be running these batch files for a while... lol

How to delete oldest files when a folder size goes above limit.

I needed to delete surveillance video files once I have more than certain amount in my hard drive. Let's say I don't want to hold more than 2GByte of video files, which will probably cover  one or two month of surveillance video data.

What I need to do with a batch file is like this:

1. Check how much video files are collected so far.
2. Compare if the total size is bigger than the size I can afford.
3. If it is under the space budget, then the batch finishes.
4. If there are more than I can afford, I want to delete oldest files first.
5. So I need to figure out what date/time is the oldest.
6. I will delete files that are at the date/time.
7. Go to the first step in order to see if I need to delete more.

This was little tricky without installing any external applications.

I had to make two separate batch files due to the syntax limitations on For loop.
One looks like this (c:\batch\delete_surveillance_find_oldest.bat):

@ECHO OFF
SET fdate=%1
SET fdate=%fdate:~6,4%%fdate:~0,2%%fdate:~3,2%
SET /a DATE_DIFF=%OLDEST_DATE% - %fdate%
IF NOT "x%DATE_DIFF:~0,1%" == "x-" SET OLDEST_DATE=%fdate%

It takes a Date value as an input argument.
It compares the date with an oldest date.
If it the date is older than the one that has been found, it updates it.

Another file is like this (c:\batch\delete_surveillance.bat):

@ECHO OFF
SET MAX_SIZE_IN_MB=1800
SET TargetDir=C:\surveillance
SET FIND_OLDEST=C:\batch\delete_surveillance_find_oldest.bat

IF NOT EXIST "%TargetDir%" GOTO :ERROR_ARG
IF NOT EXIST "%FIND_OLDEST%" GOTO :ERROR_ARG

REM SET DeleteCmd=ECHO
SET DeleteCmd=DEL /f

:RESTART
SET TotalSize=0
FOR /F "tokens=1,2,3" %%A IN (' dir /s "%TargetDir%" ^| FIND " File(s)" ') DO SET TotalSize=%%C
SET /a TotalSize=%TotalSize:,=%/1024/1024
ECHO Size limit: %MAX_SIZE_IN_MB%
ECHO Total size: %TotalSize% MB

SET /a SizeDiff=%MAX_SIZE_IN_MB% - %TotalSize%
IF NOT "x%SizeDiff:~0,1%" == "x-" GOTO :SPACE_ENOUGH

ECHO Space is not Enough...
ECHO Searching for the date of the oldest file...
SET /a OLDEST_DATE=99998877
FOR /F "tokens=1,2" %%A IN (' dir /s /a-d "%TargetDir%" ^| FIND "/" ' ) DO CALL "%FIND_OLDEST%" %%A
ECHO Oldest date found: %OLDEST_DATE:~0,4%/%OLDEST_DATE:~4,2%/%OLDEST_DATE:~6,2%

ECHO Deleting the oldest files...
FORFILES /p %TargetDir% /s /D -%OLDEST_DATE:~4,2%/%OLDEST_DATE:~6,2%/%OLDEST_DATE:~0,4% /c "cmd /c @FOR %%I IN (@path) DO @IF NOT EXIST %%~sI\NUL ( @ECHO @path & %DeleteCmd% /f @path )"
ECHO Let's see if we have enough space now.
ECHO.
GOTO :RESTART

:SPACE_ENOUGH
ECHO There are enough space.
GOTO :END

:ERROR_ARG
ECHO Input arguments or settings are incorrect.
ECHO TargetDir = %TargetDir%
ECHO FIND_OLDEST = %FIND_OLDEST%
GOTO :END

:END

This will follow the process I described above.

Recording surveillance video from Foscam IP Camera without iSpy

*Edit ( Aug/18/2014 ): I made a C/C++ program that stores the stream video from Foscam IP camera with libvlc. It may work for you better. Check out the article here.

iSpy is a cool open source software that records surveillance video. Although it has several features, the features I was using was recording video/audio when motion is detected. It had been working very well and I was enjoying it a lot.

A little problem was that this software has to be running 24/7 and all of the video data must be streamed from IP camera to the computer that iSpy is running even when there is no motions are detected. Because the motion detection happens on the computer side not on the camera side, the computer must download all the video data from the IP camera via WiFi. I figure the video data isn't too big. It was about 7.3MByte per minute, which is 122KByte/sec.

Today I found that my IP Camera (Foscam FI8910W Wireless IP Camera) has motion detection feature already in it. So it made me thinking that when IP camera detect any motions, it can let my computer know that the recording has to start. This way, the WiFi bandwidth is saved and CPU power that were supposed to be spent on image processing is saved. I don't think neither of them are big deal but I started implementing my idea for fun.

The idea was that my Foscam Camera can upload image files onto FTP server when motion is detected. But it doesn't have any ways to upload video streaming data. So I am recording video streaming data when the FTP connection is requested, while the actual image uploading from the camera will be ignored.
The working scenario is like this:

1. Foscam Camera monitors the scene.
2. Foscam Camera detect motions
3. When any motions are detected Foscam Camera try to connect to FTP server and upload image files.
4. On the FTP server side, it ignores whatever is sent from Foscam Camera.
5. But at the time FTP connection is requested, VLC is launched and VLC start recording video streaming data directly from Foscam Camera.

Settings on Foscam Camera

 "FTP Service Settings" looks like this.

• "FTP Server" is the IP number of the computer that is going store the video streaming data.
• "FTP port" is a fake number of this specific task. I will use a number "2121" for this article but you can change it for any numbers.
• "FTP User" should be "camera". It is case sensitive like most of FTP login ID.
• "FTP Password" doesn't matter for now. But you can make it count by changing the fake ftp server batch file later.
• "FTP Mode" should be "PASV" for now.

The way I am explaining here is not going to use a real FTP server. I made a fake FTP server that pretends to be a FTP server. So User/Password doesn't matter at all.

"Alarm Service Settings" looks like this:

• "Motion Detection Alarmed" should be checked.
• "Upload image on Alarm" should be checked.
• "Upload interval (seconds)" can be bigger than 60. This number is something you can tweak later. I don't recommend any number below 60.

These are all settings on Foscam Camera side.
Now when it detect any motions, the Camera will try to upload to the computer. If you have a real FTP server, it will be working. But you will get only images not audio/video.

Now I will explain computer side. This is little hacky and hard to understand but there is no way to harm the computer at all. You will need to install NetCat for windows and VLC for windows. My implementation is all based on Windows but if you are familiar with shell script, you can translate it by yourself because NetCat and VLC should be working on Linux/Mac too.

You need to install

• NetCat for Windows
• VLC

Let me show you how VLC can record video stream directly. ( This example is taken from here )

"C:\Program Files\VideoLAN\VLC\vlc.exe" http://xxx.xxx.xxx.xxx:####/videostream.asf?user=UserName&pwd=Password --qt-start-minimized --no-qt-notification --run-time=TimeInSeconds :demux=dump :demuxdump-file=MyCamera1.asf vlc://quit

Other URL commands for Foscam IP Camera is here.

Now let's have a batch file that know where to store the video stream data and where to get from.
Save this batch file as "c:\batch\record_foscam.bat"

@ECHO OFF
SET IP=xxx.xxx.xxx.xxx
SET ID=userNameForCamera
SET PWD=passwordForCamera
SET run_time_in_sec=60

REM *** PATH should not have quotation marks ***
SET VLC=C:\Program Files\VideoLAN\VLC\VLC.EXE
SET ASF_BASEDIR=C:\surveillance\video\Foscam
SET YEAR=%DATE:~-4%
SET MONTH=%DATE:~4,2%
SET DAY=%DATE:~7,2%
SET HOUR=%TIME:~0,2%
SET MIN=%TIME:~3,2%
SET SEC=%time:~6,2%
SET ASF_FILENAME=%ASF_BASEDIR%\1_%YEAR%-%MONTH%-%DAY%_%HOUR%-%MIN%-%SEC%.asf

ECHO Recording %run_time_in_sec%seconds of Video stream on to %ASF_FILENAME%...
REM "%VLC%" http://%ID%@%IP%/videostream.asf^?user=%ID%^&pwd=%PWD%^&res=8^&rate=6 --qt-start-minimized --noqt-notification --run-time=%run_time_in_sec% :demux=dump :demuxdump-file="%ASF_FILENAME%" vlc://quit

"%VLC%" http://%ID%@%IP%/videostream.asf^?user=%ID%^&pwd=%PWD%^&res=8^&rate=6 --qt-start-minimized --noqt-notification --run-time=%run_time_in_sec% --sout=#transcode{vcodec=h264,acodec=mpga,channels=1}:standard{dst="%ASF_FILENAME%"} vlc://quit

You will need to change IP, ID, PWD and ASF_BASEDIR.

• IP is the IP number of your Foscam IP Camera.
• ID is the user name for the camera that you use when you access Camera Web Interface.
• PWD is the password for the user name on the Camera.
• ASF_BASEDIR is where you want to store the video streaming data.

At this point, you should run this batch file and see if it does store the video data.
If you don't get any files saved, you will need check the id, password and VLC path.



Now here is my Fake FTP Server.
Save it as a batch file like "c:\batch\fake_ftp_server.bat"

@ECHO OFF
REM *** IP number of FTP server ***
SET ServerIP=xxx.xxx.xxx.xxx

REM *** Actual data transfer port number ***
SET ServerDataPort=2120

REM *** Case sensitive user name ***
SET ValidUserID=camera

REM *** Where is the batch file that triggers VLC to record video stream ***
REM *** Do no use quotation marks on the path
SET VLC_CAPTURE_BAT=C:\batch\record_foscam.bat


REM *** Do not change below ***
SET /a ServerPortHigh=%ServerDataPort% / 256
SET /a ServerPortLow=%ServerDataPort% - %ServerPortHigh% * 256

ECHO 220 Welcome message

SET /p user=
PAUSE > NUL
IF NOT "x%user%" == "xUSER %ValidUserID%" GOTO :ERROR_AUTH
ECHO 331 Please specify the password.

SET /p pass=
PAUSE > NUL
ECHO 230 Guest login ok, access restrictions apply.

SET /P typei=
PAUSE > NUL
ECHO 200 Type set to I

SET /P pasv=
ECHO 227 Entering passive mode (%ServerIP:.=,%,%ServerPortHigh%,%ServerPortLow%)
PAUSE > NUL

SET /P stor=
ECHO 150 FILE: no file will be stored but video recording will start...START /B CMD /C CALL "%VLC_CAPTURE_BAT%" 2> NUL > NULECHO 226 Transfer complete.
PAUSE > NUL

SET /P next=
PAUSE > NUL
GOTO :END

:ERROR_AUTH
ECHO 530 Invalid login information...
GOTO :END

:END

Once you saved this batch file somewhere, you will need to modify two variables: ServerIP, ServerPort.

• ServerIP is the IP number of the computer.
• ServerDataPort needs little explanation. When FTP client access to a FTP server, it makes two connections. One is for the command communication and the other is for actual data transporting. They are traditionally port number 21 and 20 respectively. In this article, I am using 2121 and 2120 respectively. If you scroll up, you can see "FTP port" is set to be 2121 on the Foscam Camera and this fake_ftp_server is going to listen to the port number. Then this fake FTP server has to tell what port number is for the data transfer. So the ServerDataPort should be any port number that is not in use. In this article I will use 2120 for the data transfer.

Remember that this FTP server is a fake FTP server. It can check user name and password but for now it cares user name but not password.


The last step of this article is to have a network port listener.
NetCat will keep listening any network access from other machines.
Here is the batch file and save it as "c:\batch\fake_ftp_listener.bat"

@ECHO OFF
SET FTP_Port=2121
SET FTP_DataPort=2120

REM *** Do not use quotation marks on path
SET FTP_SERVER=C:\batch\fake_ftp_server.bat
SET NETCAT=C:\batch\nc111nt\nc.exe

ECHO [%DATE% %TIME%] Starting fake FTP server for IP camera...

:RESTART
START /B CMD /C "%NETCAT%" -l -p %FTP_DataPort% ^> NUL
"%NETCAT%" -l -p %FTP_Port% -e "%FTP_SERVER%"
"%NETCAT%" -z localhost %FTP_DataPort%
GOTO :RESTART

You will need to set the Path of batch file and NetCat; FTP_SERVER and NETCAT respectively.
Make sure the Port numbers are matching to other batch files; FTP_Port and FTP_DataPort.

Once you start this batch file, NetCat will start listening two network ports.
Now try to run the batch file.
To test whether it works or not, you can manually connect to the port while fake ftp listener is running.

telnet localhost 2121

If you see a welcome message, type in "USER camera".
If it asks password, then it is working.

As an optional step, you can start it as a hidden process when the windows start.
Save this line as a visual basic script file, "c:\batch\invisible.vbs"

CreateObject("Wscript.Shell").Run """" & WScript.Arguments(0) & """", 0, False

 Save this line as a batch file, "c:\batch\fake_ftp_listener_hidden_start.bat"

@"C:\Windows\System32\wscript.exe" "C:\batch\invisible.vbs" "C:\batch\fake_ftp_listener.bat"

 Now you can make a shortcut of the "fake_ftp_listener_hidden_start.bat" in start up folder of your windows.
It is usually C:\Users\[YourUserName]\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup

Generating Collada loader with JAXB

Collada is a XML 3D Model data format. I was looking for a library that can automatically load in structure. I found JAXB that does exactly what I was looking for. However, there was a little trouble generating Java source code.

After struggling for a few hour, I was able to generate JAXB source codes from Collada Schema v1.5.
Here is how I did and it is for anybody who may have the problem I was having.

I got the Collada schema file from here.
At my first try, I got errors:

> xjc.exe -p collada collada_schema_1_5.txt
[ERROR] Property "Source" is already defined. Use <jaxb:property> to resolve this conflict.
[ERROR] The following location is relevant to the above error

This error is due to name conflict. The name "source" is defined for an attribute and inner element. It can be like this:

<skin_type source="somekindofURL"> <source></source> </skin_type>

 The generated Java source code from JAXB will have a member method "getSource()" for both the element and the attribute so does the conflict happens.

This issue can be resolved by providing "binding" information that simply rename one of them from "source" to something else. The binding information looks like this:

<bindings br="" xmlns="http://java.sun.com/xml/ns/jaxb">          xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance"
          xmlns:xs="http://www.w3.org/2001/XMLSchema"
          version="2.1">
    <bindings schemalocation="collada_schema_1_5.xsd" version="1.0">

        <!-- rename the value element -->
        <bindings node="//xs:complexType[@name='skin_type']">
            <bindings node=".//xs:attribute[@name='source']">
                <property name="source_attr">
            </property></bindings>
        </bindings>
       
        <bindings node="//xs:complexType[@name='morph_type']">
            <bindings node=".//xs:attribute[@name='source']">
                <property name="source_attr">
            </property></bindings>
        </bindings>

    </bindings></bindings>

But this solves only half of the problem.

> xjc.exe -b collada_binding.xjc -p collada collada_schema_1_5.xsd
[ERROR] Property "MiOrMoOrMn" is already defined. Use <jaxb:property> to resolve this conflict.

 This error was easier to resolve by internet searching.
https://jaxb.java.net/guide/Dealing_with_errors.html
https://weblogs.java.net/blog/kohsuke/archive/2006/03/simple_and_bett.html

This issue can be solved by having an extension file:

<xs:schema br="" targetnamespace="no-such-thing">    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" jaxb:version="2.0"
    xmlns:xjc= "http://java.sun.com/xml/ns/jaxb/xjc" jaxb:extensionBindingPrefixes="xjc">
  <xs:annotation>
      <xs:appinfo>
        <jaxb:globalbindings>
            <xjc:simple>
        </xjc:simple></jaxb:globalbindings>
      </xs:appinfo>
  </xs:annotation></xs:schema>

 Now JXB should be able to generate the Java source files from Collada schema.

> xjc.exe -extension simpleMode.xjc -b collada_binding.xjc -p collada collada_schema_1_5.xsd

 I am not sure what kind of magic this extension file is playing but it works for me for now...

McAfee Online Backup

Yesterday I purchased McAfee Online Backup service.
I think it was a good deal; *unlimited* online backup storage for $60/year.

Their webpage didn't give me much information about their service. I was wondering what kind of options I can tweak, how many computers I can use, how can I restore files, how long they will keep my data after my license expires and so on.

What I found after buying it are:

1. $60/year is only for one computer. I need to add another $60/year for another computer but it seems that they give some discount from 3rd computer.
2. They delete every backup data as soon as my license expires.
3. Since my internet uploading speed is not as fast as downloading speed, it seems to take for a while to consume big amount of promised backup storage.
4. One license covers only one computer to be using the backup software. But it seems that I can download any backup files from any other computers via web user interface. I think this is very useful and otherwise I will be complaining a lot.

The Backup Options:
The backup software runs as a windows tray icon. It doesn't give me complex tweaks. Basically it lets me select which folders to be backed up. It also has some network traffic controlling and CPU utilization checkup to be a less intrusive background task.

The Web Restore:
A picture above is take from their web restore user interface. I can select files to download, then the system generate a ZIP file that includes the files. It seems to take some time to generate the file, which is understandable; when the data size is big, the system will have to spend some time to compress them. The web user interface doesn't seems to allow me to download more than 20GB at once, which is a minor limitation for me.

The Price:
My internet uploading speed is about 0.1MByte/s, which means I can upload 8GByte/day and 3TByte/year. So assuming that I have lots of data to backup 24/7, this is a good deal. Because it is like I am paying $60 for 3TB for the first year and in the next year I will be paying another $60 for another 3TB top of the 3TB I used the previous year.

Comparing to HDD price, I think this service is a good deal although I thought that $60 is little too much at first thought. These days the HDD price is around $90~$140 for 1~3TByte. Although HDD last more than just one year, most of time the limited-warranty is 1/2 years. After 3years later, HDD may stop working and there is a risk to lose all the back up data.

The real question for me at this point is whether I actually have data to backup that much unless I want to securely backup all the HD movie files.

*PS: I tried to access Network Mounted Drive from McAfee Backup software but it didn't show the drive at all. I believe there gotta be a way to trick the software and allow me to access remote files without another license.

Java lamda expression in version 8 next year.

Java will have Lamda expression on the coming new version 8 next year. This info is from the official java web site: here

This anonymous class feature was from the day one.

printMembers(
    roster,
    new CheckMember() {
        public boolean test(Member p) {
            return p.getGender() == Member.Sex.MALE
                && p.getAge() >= 18
                && p.getAge() <= 25;
        }
    }
);

But with the new Lamda expression it will become like this:

printMembers(
    roster,
    (Member p) -> p.getGender() == Member.Sex.MALE
        && p.getAge() >= 18
        && p.getAge() <= 25
);

Here is how the magik works.
The interface "CheckMember" should have only one member method. The interface that has only one member method is called "functional interface".

interface CheckMember {
    boolean test(Member p);
}

Because functional interface has only one member method, we can omit the obvious function name.

iTunes auto sync over Wifi fixed..

I have been having a trouble on auto sync feature between my iTunes and my iPhone.
When I was searching Google, many people fixed the same problem by de-selecting "prevent iPods,iPhones, and iPads from syncing automatically" or selecting "Sync with this iPhone over Wi-Fi".

But it didn't fix my problem. One of guys claimed that he fixed the issue by clean uninstalling iTunes with registry cleaner. It didn't work for me too.

Today I fixed the problem. My problem was on Bonjour. It was installed at "C:\Program Files (x86)\Bonjour" and it didn't go away even after I uninstalled every Apple applications. It turned out that it didn't go away because one of DLL files in the folder was still in use by AirDisplay.

Deleting DLL was little tricky because Windows doesn't allow users to alter a file while it is open by any processes. Somehow there were more than AirDisplay was using the Bonjour DLL. I opened CMD window and renamed the DLL in the Bonjour folder and delete the folder. I restarted Windows7 and re-installed iTunes and AirDisplay.

Mileage will vary but this way fixed my problem and I hope this info can help others too.

Reusable Logic Flow code can be separated from platform depended code implementation

Yesterday I came up with an interesting idea about programming language.
The idea is from a question how I can separate detail code implementation from abstract logic.

For example, Java introduced a powerful concept called Interface. An interface is nothing but a collection of method signature. It cannot contain any implementation or any variables. First time when I saw it, I thought it is useless because it does nothing for me without any implementation. But the important aspect of interface was to separate the abstract layer of programming code from detail implementation. It also allowed us to reuse the abstract layer a lot more and also allowed us to replace old implementation without affecting other code that are relying on the interface.


I didn't extend my idea from interface but later I found the similarity. And it is easier to understand from the idea of Interface. A function is usually mixed with Logic and implementation details. Let me show us a code example first. This code is taken from a random Google search: http://www.programmingsimplified.com/c-program-copy-file

#include "stdio.h"
#include "stdlib.h"
 
int main()
{
   char ch, source_file[20], target_file[20];
   FILE *source, *target;
 
   printf("Enter name of file to copy\n");
   gets(source_file);
 
   source = fopen(source_file, "r");
 
   if( source == NULL )
   {
      printf("Press any key to exit...\n");
      exit(EXIT_FAILURE);
   }
 
   printf("Enter name of target file\n");
   gets(target_file);
 
   target = fopen(target_file, "w");
 
   if( target == NULL )
   {
      fclose(source);
      printf("Press any key to exit...\n");
      exit(EXIT_FAILURE);
   }
 
   while( ( ch = fgetc(source) ) != EOF )
      fputc(ch, target);
 
   printf("File copied successfully.\n");
 
   fclose(source);
   fclose(target);
 
   return 0;
}

Although the program itself is short and easy to understand, the issue is that functional logic flow and implementation details are mixed.This problem may not be obvious for short program like this. But when we want the program to be running on multiple platforms, we have to replace implementation details.

First remedy for multi-platform support is usually "#ifdef PLATFORM_A / #else / #endif".
It gets harder to read and gets longer to read.

The point of my view is that even when the implementation details can vary, the logic flow should stay.

After I apply my idea the code becomes like this:

class ICopyFileLogic
{
public:
 virtual void GetFilenameToCopy();
 virtual void OpenSourceFile();
 virtual bool IsSourceFileOpen();
 virtual void NoticeOpenFailed();
 virtual void GetFilenameToCreate();
 virtual void OpenTargetFileToWrite();
 virtual bool IsTargetFileOpen();
 virtual void NoticeTargetOpenFailed();
 virtual bool GetOneByte();
 virtual void WriteOneByteOnTargetFile();
 virtual void NoticeCopySuccessful();
 virtual void CloseFileHandles();
};
 

int _tmain(int argc, _TCHAR* argv[])
{
 ICopyFileLogic* cf = new PCopyFile_Win32();

 cf->GetFilenameToCopy();
 cf->OpenSourceFile();
 
 if( !cf->IsSourceFileOpen() )
 {
  cf->NoticeOpenFailed();
  exit(EXIT_FAILURE);
 }

 cf->GetFilenameToCreate();
 cf->OpenTargetFileToWrite();
 
 if( !cf->IsTargetFileOpen() )
 {
  cf->NoticeTargetOpenFailed();
  exit(EXIT_FAILURE);
 }
 
 while( cf->GetOneByte()  )
  cf->WriteOneByteOnTargetFile();

 cf->NoticeCopySuccessful();
 cf->CloseFileHandles();
 return 0;
}

class PCopyFile_Win32 : public ICopyFileLogic
{
 char ch, source_file[20], target_file[20];
 FILE *source, *target;

 void GetFilenameToCopy()
 {
  printf("Enter name of file to copy\n");
  gets(source_file);
 }

 void OpenSourceFile()
 {
  source = fopen(source_file, "r");
 }

 bool IsSourceFileOpen()
 {
  return source != NULL;
 }

 void NoticeOpenFailed()
 {
  printf("Press any key to exit...\n");
 }

 void GetFilenameToCreate()
 {
  printf("Enter name of target file\n");
  gets(target_file);
 }

 void OpenTargetFileToWrite()
 {
  target = fopen(target_file, "w");
 }

 bool IsTargetFileOpen()
 {
  return target != NULL;
 }

 void NoticeTargetOpenFailed()
 {
  fclose(source);
  printf("Press any key to exit...\n");
 }

 bool GetOneByte()
 {
  return ( ch = fgetc(source) ) != EOF;
 }

 void WriteOneByteOnTargetFile()
 {
  fputc(ch, target);
 }

 void NoticeCopySuccessful()
 {
  printf("File copied successfully.\n");
 }

 void CloseFileHandles()
 {
  fclose(source);
  fclose(target);
 }
};
  

When the idea was in my head, it was more interesting than after I wrote them down. The new version of the code is longer and also it utilizes stack memory less than before, which I think not preferable. It will also prevent the compiler from optimizing harder.

But the good part is, as I discussed earlier, the logic flow is reusable so that we can replace the platform dependent code implementation without affecting the logic flow. The "main" function doesn't even need to include stdio.h or stdlib.h.

I may need to find a better example than this to convince anybody else of the usefulness of the idea.